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ANALYSIS OF TRENDS FROM DOD-LEVEL PEER-REVIEWED 

CONTRACTS 


ABSTRACT 


Department of Defense (DOD) spending has been steadily increasing ever since the early 
1990s. During that period, the acquisition workforce has steadily declined. This situation 
resulted in an undermanned and undertrained contracting workforce with an increased 
workload. With the workforce spread thin, lapses in contracting processes occurred. As a 
result of these issues, in 2008, the DOD established the requirement for independent 
management reviews, or peer reviews, of contractual actions. 

Since the onset of the peer-review requirement, the Defense Procurement 
Acquisition Policy (DPAP) has maintained a database of peer-review results. Data 
analytics were used to analyze the frequency of occurrences of the data elements within 
the DPAP database of peer-review results in an effort to answer two research questions. 
First, are there trends within the peer-review results of DOD-level peer-reviewed 
contracts? Second, are any trends identified related to the competency gaps identified in 
the 2007 Department of Defense Contracting Workforce Competency Assessment Final 
Report ? 

Trends within the data elements present in the DPAP database of peer-review 
results were identified. Certain categories garnered more attention of the peer-review 
teams. Based on trends identified, recommendations are provided to improve the overall 
usefulness of the DPAP database of peer-review results. 
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I. 


INTRODUCTION 


A. BACKGROUND 

Department of Defense (DOD) spending has been steadily increasing ever since 
the early 1990s. During the same time period, the acquisition workforce has steadily 
declined. The 2007 Gansler Commission report on Army contracting noted that over a 
12-year period, contracting actions had risen 350% while the Army contracting 
workforce had been reduced by 50% (Gansler, 2007). This situation resulted in an 
undermanned and undertrained contracting workforce, with a vastly increased workload. 
Within the DOD, workload per GS-1102 (Contract Specialist) staff member rose from 
an annual average of $6.4 million in contract actions in fiscal year (FY) 1996 to nearly 
$13 mi llion in FY 2005 (Girovasi, 2007). 1 With the onset of overseas contingency 
operations in Iraq and Afghanistan, demand for the acquisition workforce spiked. The 
increase in spending, along with the corresponding decrease in the acquisition workforce, 
is highlighted in Figure 1. 


1 As the dollar value of a contract action increases, so does the workload required to process it. For 
example, micropurchase procedures apply under $3,000, and simplified acquisition procedures apply from 
$3,000 up to $150,000 (FAR 13.003(b)(1), 2014). Additional workload is generated when a contractor is 
required to submit Certified Cost or Pricing Data for purchases above $700,000 when no exceptions apply 
(FAR 15.403-4(a)(l), 2014). 
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Fiscal Year 

■ DoO Procurement Budget — Acquisition Organization Workforce 

Figure 1. DOD Acquisition Trends (from Gansler, 2007) 

With the workforce spread thin, lapses in contracting processes began to occur. A 
Government Accountability Office (GAO) report, GAO-05-274, Contract Management: 
Opportunities to Improve Surveillance on DOD Service Contracts , issued in March 2005, 
reviewed 90 service contracts. Out of these 90 contracts, the GAO (2005) found 
insufficient surveillance on 24. From those 24 contracts, 15 had no surveillance 
whatsoever (GAO, 2005). The Gansler Commission report revealed that, as of 2007, 
83 Army criminal investigations relating to contract fraud were ongoing in Iraq, 
Afghanistan, and Kuwait. At the same time, 23 government employees (both civilian and 
military) were charged or indicted in federal court. The contracts affected represented 
over $6 billion in value (Gansler, 2007). 

As a result of these and other issues discussed during the literature review chapter, 
the DOD established the requirement for independent management reviews, or peer 
reviews, of contractual actions in 2008. The purpose of the peer-review requirement was 
to help address the shortfalls in manning and experience within the DOD contracting 
community by ensuring policy and regulations were followed in a consistent manner, and 
that the available experience was shared across the contracting workforce (Assad, 2008). 
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Identifying and tracking the trends within the data provided by the peer-review process is 
critical to identifying systemic problems within both the training of the DOD’s 
contracting workforce and how the contracting workforce is operating. 

B. PURPOSE 

The primary purpose of this project was to examine the Defense Procurement 
Acquisition Policy (DPAP) database of peer-review results and identify any trends within 
the data. This project was not intended to be an exhaustive study of the DPAP database of 
peer-review results. Rather, it was contemplated as a way to identify overarching trends 
for further analysis in the future. There is one secondary goal associated with this project: 
examine any relationships between trends observed within the DPAP database of peer- 
review results and the competency gaps by the Contracting Competency Model (CCM) in 
the Department of Defense Contracting Workforce Competency Assessment Final Report 
(DPAP, 2007). 

C. RESEARCH QUESTIONS 

In an effort to investigate a previously unexplored topic, I crafted two research 
questions: 

• Are there trends within the peer-review results of DOD-level peer- 
reviewed contracts? 

• Are any trends identified related to the competency gaps identified in the 
Department of Defense Contracting Workforce Competency Assessment 
Final Report (DPAP, 2007)? 

D. BENEFITS AND LIMITATIONS OF THIS PROJECT 

The results of this project can be used to target and strengthen areas of training 
that are leading to systemic issues within the contracting process. It can also be used to 
identify areas of the contracting process that are simply being neglected and would 
benefit from additional emphasis by contracting officers and their supervisors. Improved 
understanding of the trends within the DPAP database of peer-review results will enable 
policymakers to structure decisions appropriately to increase both the efficiency and 
effectiveness of the DOD contracting workforce. 
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The limitation of this project rests with the fact there were only 288 entries within 
the DPAP database of peer-review results. Thus, it was not possible to obtain a larger 
population or sample size to analyze. 

E. RESEARCH METHODOLOGY 

The research began with a literature review. The literature review addresses why 
DOD conducts peer reviews and addresses corporate procurement review practices, 
development of the peer-review requirement, and guidance on conducting peer reviews. 
I approached the research questions by analyzing the DPAP database of peer-review 
results in terms of frequency of occurrences of various data aspects, including Category, 
Type of Contract, Review Phase, and Type of Feedback. I examined the Narrative 
comments section from a qualitative standpoint to identify any recurring themes within 
the comments. 

F. ORGANIZATION OF REPORT 

The report consists of five chapters. Chapter I contains a brief background, the 
purpose of the research, research questions examined, relative benefits and limitations of 
the project, and an overview of the research methodology. Chapter II is the literature 
review, which is primarily concerned with addressing the questions of why we peer 
review contractual actions, how the requirement was developed, and how it has been 
implemented. To that end, the history of the Defense Acquisition Workforce 
Improvement Act (DAWIA) is examined. I review multiple GAO and commission 
reports that reveal the decline in the experience and manning of the acquisition 
workforce, coupled with the rise in spending activities and the resultant issues that 
caused. Chapter III discusses where the data originated and how it was analyzed. Chapter 
IV provides the results from the analysis of the data. Chapter V contains the summary of 
the project, recommendations for improving the DPAP database of peer-review results, 
and areas for further research relating to the findings of this project. 
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G. SUMMARY 

Chapter I provided the background information on the peer-review requirement 
and the purpose of the project, stated the research questions and the benefits and 
limitations of the project, and briefly addressed the methodology and organization of the 
written report. The next chapter contains the literature review. The literature review 
addresses why the DOD conducts peer reviews of contractual actions. It also addresses 
corporate procurement review practices, development of the peer-review requirement, 
and guidance on conducting peer reviews. 
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II. LITERATURE REVIEW 


A. INTRODUCTION 

The previous chapter discussed the basis of this research. In this chapter, I 
examine why the DOD conducts peer reviews of contractual actions. I also examine 
development of the peer-review requirement, and guidance on conducting peer reviews. 
Elements of this chapter include explanations of the DOD’s initiation of required peer 
reviews, the definition of a peer review, and the execution of peer reviews. Additionally, 
I review how both public and private procurement agencies evaluate performance. 

B. WHY PEER REVIEW? 

1. Introduction 

To understand the reasoning behind the peer-review requirement, it is necessary 
to look back to the beginnings of the modem defense acquisition workforce, which can 
be traced back to the early 1990s, with the passage of the Defense Acquisition Workforce 
Improvement Act (DAWIA, 1990). The history since 1990 develops a story of the 
development of a truly professional workforce and how it ran into issues of cutbacks in 
an era of persistent defense spending cuts. This led to a situation of neglect within the 
acquisition workforce in a time of persistent conflict when the workforce could have been 
utilized to its fullest, ultimately causing the workforce to break down under the strain of 
too many requirements and too few trained personnel. 

2. Defense Acquisition Workforce Improvement Act 

As a result of an acquisition workforce that was plagued by scandal and 
inefficiencies, the DAWIA was developed and enacted into law in 1990. Fraud, waste, 
and abuse have gone hand in hand with military operations throughout history. The U.S. 
government’s response after each conflict has been to enact progressively more 
restrictive legislation in the name of acquisition reform. Defense historian William 
Gregory (1989) described in his work The Defense Procurement Mess the state of 
acquisitions in the late 1980s “as one that had been managed and over-reformed into 
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impotence with volumes of oversight regulations.” Scandals dragged the issues further 
into the spotlight. Operation Ill Wind, a Department of Justice probe that uncovered 
widespread corruption and incidents of fraud and bribery within the defense procurement 
system, was concluded in 1988. In all, the operation resulted in more than 60 convictions, 
including that of former Assistant Secretary of the Navy Melvin Paisley (Layton, 2007). 

Voted into law as Public Law 101-510 on November 5, 1990, the DAWIA was 
codified in Title 10, Chapter 87 of the U.S. Code and amended in 1991, 1993, 2001, 
2003, 2008, 2011, and 2013 (DAWIA, 2013). The purpose of the DAWIA was to shift 
the focus from regulating acquisition procedures to developing a professional acquisition 
workforce. It established a certification process for contracting as a career field, to 
include establishing standards for education, training, and progressive levels of 
experience. DAWIA also prescribed the creation of a center for defense acquisition 
education. The Defense Acquisition University (DAU) was formed in 1992 to satisfy that 
requirement. The DAU was initially established from a consortium of existing DOD 
organizations, bound together by a memorandum of agreement to cooperate as the DAU. 
The DAU has been highly successful in raising professional standards of education 
for acquisition professionals; however, while quality has increased, quantity has not 
(Layton, 2007). 

3. The Acquisition Workforce in Decline 

Much has been written over the past six years about issues facing the acquisition 
workforce. The Acquisition Advisory Panel (AAP) issued a report in 2006 through the 
Office of Federal Procurement Policy (OFPP) detailing the challenges facing the 
acquisition workforce. The acquisition career field was not spared from the larger defense 
drawdown of the early to mid-1990s. Actual numbers for the time period are difficult to 
find, for two main reasons. First, each organization in the federal government defines 
“acquisition professional” differently; and second, prior to 1999, there was no 
requirement mandating tracking acquisition professionals separately within the federal 
employment system. What is known is that the acquisition workforce went through 
substantial reductions throughout the 1990s, and little to no hiring took place. 
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Meanwhile, the workforce continued to gain experience while it edged ever closer to 
retirement (OFPP, 2007). Figure 2 illustrates the issue, with a large number of contracting 
officers with less than 10 years of experience, with a dip in the mid-level experience 
range, followed by a large number of employees at the retirement age. 


Defense Acquisition Workforce - DOD-Wide 

Distribution by Years to Retirement Eligibility (Civilians)(FY09) 


22.862 or 37.5% of the 
civilian members of the defense 
acquisition workforce wl II 
become eligible to retire 
with full benefits before 2020. 
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19,395 or 16% of the civilian 
members of the defense acquisition 
workforce are eligible 
to retire with full benefits. 
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Figure 2. Defense Acquisition Workforce Retirement Eligibility Distribution 

(from DOD, 2010, p.2-22) 


Since 1999, the differences across the federal government in defining acquisition 
professionals make it difficult to ascertain precise statistics about growth trends within 
the workforce, though all numbers point toward decline, ranging anywhere from 3% to 
27% (OFPP, 2007). Within the current acquisition workforce, 50% were retirement 
eligible in 2010 (Girovasi, 2007). Another well-documented fact is the increase in 
acquisition-related spending since 1990. Paradoxically, as the acquisition workforce was 
reduced, the reliance on the workforce increased. Consider some examples: 

• Federal acquisition expenditures tripled from FY 1991 to FY 2006, 
reaching a level of $424 billion. 
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• Federal acquisition spending increased by 65% from FY 2001 to FY 2005, 
representing an increase from $235 billion to $388 billion. 

• Within the DOD, workload per GS-1102 (Contract Specialist) staff 
member rose from an annual average of $6.4 million in contract actions in 
FY 1996 to nearly $13 mi llion in FY 2005. 

• Contracts for services accounted for 60% of total spending in FY 2005 and 
2006 (Girovasi, 2007). 

The issue at hand becomes one of more contract actions to process with fewer, 
less-experienced contracting officers available to process them. The 2007 Gansler 
Commission report on Army contracting noted that over a 12-year period, contracting 
actions had risen 350% while the Army contracting workforce had been reduced by 50% 
(Gansler, 2007). 

4. The System Breakdown 

Several reports, documents, and commissioned studies have established what 
happened once the contracting system was placed under stress. GAO-02-737 was issued 
in July 2002, approximately 10 years after the overhaul of the acquisition system began, 
but before the United States’ involvement in Iraq and Afghanistan and the surge in 
contracting actions. The GAO (2002) noted that when the DOD had adopted a definition 
of the acquisition workforce, it was very multidisciplinary in nature, covering various 
functions of the contracting team. Various government civilian agencies, by contrast, had 
not taken such a broad approach and only considered the GS-1102 series to make up the 
contracting team. As a result of the DOD outlook, they had developed broad training and 
tracking programs to assist in maintaining and accounting for their acquisition 
professionals. The GAO (2002) found overall that the DOD was executing the DAWIA 
reforms well; however, the government civilian agencies still had work to do. The report 
also found, through the interview process, that acquisition leadership felt that funding 
was currently adequate to maintain training proficiency levels, but that leadership worried 
about upcoming budget cuts, particularly to the DAU’s budget (GAO, 2002). 

By 2005, the situation seemed to have worsened. GAO-05-274, Contract 
Management: Opportunities to Improve Surveillance on DOD Service Contracts, issued 
in March 2005, reviewed 90 service contracts. Out of these 90 contracts, the GAO found 
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insufficient surveillance on 24. From those 24 contracts, 15 of them had no surveillance 
whatsoever. The DOD readily acknowledged the lapse, and in response to the findings 
stated they simply did not have enough qualified contracting personnel to fulfil the 
requirement (GAO, 2005). 

Testimony by David M. Walker, Comptroller General of the U.S., before 
the House of Representatives Subcommittee on Defense and the Committee on 
Appropriations in September 2006, was captured in GAO-06-800T (Walker, 2006). His 
testimony was based on six years of research using the generally accepted government 
auditing standards (GAGAS). The work revealed that dollar values for major weapon 
system and service contracts were indeed on a sharp rise, and the workforce had 
remained flat. Issues of insufficient oversight of contractors, weak business practices, and 
poor incentives for contractors to perform well were also identified (Walker, 2006). 
Figure 3 illustrates the issue: the line within the figure demonstrates the acquisition 
workforce, while the columns represent the increase in obligations. 


f GAO 

DOD Contract Obligations and Acquisition Workforce 
Trends Since Fiscal Year 2000 


Dollars r biaons’ 
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Workforce: GAO analysis of OPM data of 14 acquisitian -related job series. 4 


Figure 3. DOD Contract Obligations and Acquisition Workforce Trends 

(from GAO, 2006, p. 4) 
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In late 2006, the GAO also convened a forum of experts on acquisition and 
management from government, academia, and the public sector to discuss overall federal 
acquisition challenges over the longer term. GAO-07-45SP found that leaders among the 
various federal agencies were not recognizing the important role the acquisition 
workforce played in their organizations. It also upheld DOD contract management as a 
federal government high-risk area, a status it had held since 1992 (GAO, 2007). 

As stories of fraud, waste, and abuse, coupled with rampant spending in Iraq, 
Afghanistan, and Kuwait, continued to mount, the Secretary of the Army established a 
commission to examine expeditionary contracting in 2007. The commission was headed 
by Dr. Jacques S. Gansler, former Undersecretary of Defense for Acquisitions, 
Technology & Logistics. The Gansler Commission found, as several previous GAO 
reports show, that the contracting workforce had become short-staffed and undertrained. 
It also highlighted massive amounts of fraud relating to government contracts, mostly in 
Iraq. As of October 23, 2007, 83 Army criminal investigations relating to contract fraud 
were ongoing in Iraq, Afghanistan, and Kuwait. At the same time, 23 government 
employees (both civilian and military) were charged or indicted in federal court. The 
contracts affected represented over $6 billion in value (Gansler, 2007). 

Further testimony before Congress by David M. Walker was captured in two 
additional GAO reports, GAO-07-1098T and GAO-08-621T (GAO 2007; 2008). These 
reports continued to highlight issues of shortages in the workforce and insufficiently 
trained acquisition personnel. Additionally, one report noted the dramatic increase in the 
role of service contracts within the DOD and a shift away from the utilization of 
government employees. Concerns were also raised about the line becoming blurred 
between contractor involvement with inherently governmental functions. The marked 
increase in the number of service contracts only multiplied the stress on the government 
contracting workforce (GAO, 2008). A report delivered to Congress in 2007 by John P. 
Hutton, Director of Acquisition and Sourcing Management, reinforced this by revealing 
that the DOD had awarded service contracts for security guard services at 57 domestic 
bases, and that 46 of them were awarded on a sole-source basis. The cost of the contracts 
rose by 25% compared to when the same contracts were previously awarded 
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competitively. The same report identified that, indexed to 2006 dollars, the amount 
awarded for DOD service contracts rose from $85.1 billion in 1996 to $151 billion in 
2006, representing a 78% increase (Hutton, 2007). 

Of specific mention, the GAO issued the report titled Status of DOD's 
Implementation of Independent Management Reviews for Service Acquisitions in January 
2010. Overall, the report found that there are still issues with the implementation, but the 
DOD is making progress. The report noted two major issues. First, at the time of the 
report, the military departments stated that they had undertaken hundreds of reviews; 
however, they could not deliver precise numbers because of a lack of a defined reporting 
process. Second, and perhaps equally grievous, the DOD had not yet developed a 
methodology or mechanism to report review results and lessons learned back to the force 
(GAO, 2010). The following section addresses the development of the peer review- 
requirement. 

C. DEVELOPMENT OF PEER-REVIEW REQUIREMENT 

The development of the peer-review requirement began in 2008 with a DOD 
memorandum informing the service branches that contractual actions over a certain dollar 
threshold would be subject to a DOD-level peer-review process, both pre- and post-award 
of the contract. It also instructed the respective services to establish internal peer-review 
processes based on lower dollar thresholds. The DFARS and Army Federal Acquisition 
Regulation Supplement (AFARS) (2014) provided specific guidance about the 
implementation of the new policy, and contracting-specific guides such as the Contract 
Attorney’s Desk Guide and the Army Contracting Command Desk Book provided 
references to the new regulations. 

1. DOD Peer-Review Implementation Memorandum 

The DPAP organization is the policy arm of DOD for defense procurement and 

acquisitions. This organization is responsible for “all Contracting and Procurement policy 

matters including e-Business in the DOD. DPAP executes that policy through the timely 

update of the DFARS, PGI, and 5000.1&2” (DPAP, 2014). The DOD established the 

requirement for contract peer reviews by issuing a policy memorandum titled “Peer 
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Reviews of Contracts for Supplies and Services” dated September 29, 2008 (Assad, 
2008). The memorandum stated three primary objectives: (1) to ensure that Contracting 
Officers across the Department are implementing policy and regulations in a consistent 
and appropriate manner; (2) to continue to improve the quality of contracting processes 
across the Department; 3) to facilitate cross-sharing of best practices and lessons learned 
across the Department (Assad, 2008). 

The peer-review requirement grew from two sources. First, the DOD had 
implemented a pilot program to evaluate proposed contract awards, focusing on the pre¬ 
contract award process. Second, Congress enacted Section 808 of the National Defense 
Authorization Act (NDAA) for Fiscal Year 2008, Public Law 110-181. This law required 
the DOD to establish requirements for post-award independent management reviews of 
contracts for services, and for sharing lessons learned from those reviews (NDAA, 2008). 
The existing peer-review pilot program expanded to satisfy the requirement brought 
about by the NDAA of 2008. 

Rather than being regulatory in nature, the peer reviews were envisioned as a 
quality control and an advisory tool for contracting officers. It is important to note that 
the agency managing the procurement still has the final decision on how the procurement 
is executed, though the memorandum did establish a requirement that all peer-review 
recommendations and their dispositions be documented in the contract file. 

The DPAP office has responsibility for organizing review teams and facilitating 
pre-award peer reviews for all contracts valued at or above $1 billion, and post-award 
reviews of service contracts valued at or above $1 billion. The contract value should also 
include the estimated value of any options associated with the contract. The DPAP-level 
peer-review teams consist of senior-level DOD contracting officials, both civilian and 
military, and members of the Office of General Counsel. 

Further, the initial memorandum directed that each military department, defense 
agency, and DOD field activity would publish its own policies for the conduct of pre- and 
post-award peer reviews for contracts valued at less than $1 billion. 
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2. Defense Federal Acquisition Regulation Supplement Peer-Review 
Requirements 

The DOD codified the requirement into the Defense supplement to the Federal 
Acquisition Regulation (FAR). The FAR is a detailed federal regulation that prescribes 
how the federal government will procure supplies, services, and equipment. The DOD 
has its own supplement to the FAR, as does each service component (Army, Navy, and 
Air Force). Defense Federal Acquisition Regulation (DFAR) Part 201.170 explains the 
requirement for peer reviews: 

201.170 Peer Reviews. 

(a) DOD peer reviews. 

(1) The Office of the Director, Defense Procurement and Acquisition 
Policy, will organize teams of reviewers and facilitate peer reviews for 
solicitations and contracts, as follows using the procedures at PGI 
201.170— 

(i) Pre-award peer reviews for competitive procurements will be 
conducted in three phases for all solicitations valued at $1 billion 
or more; 

(ii) Pre-award peer reviews for noncompetitive procurements will 
be conducted in two phases for new contract actions valued at 
$500 million or more; and 

(iii) Post-award peer reviews will be conducted for all contracts for 
services valued at $1 billion or more. 

(2) To facilitate planning for peer reviews, the military departments and 
defense agencies shall provide a rolling annual forecast of acquisitions that 
will be subject to DOD peer reviews at the end of each quarter (i.e., March 
31; June 30; September 30; December 31), to the Deputy Director, 
Defense Procurement and Acquisition Policy (Contract Policy and 
International Contracting) via e-mail to osd.pentagon.ousd-atl.mbx.peer- 
reviews@mail. 
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(b) Component peer reviews. The military departments and defense agencies shall 
establish procedures for— 

(1) Pre-award peer reviews of solicitations for competitive procurements 
valued at less than $1 billion; 

(2) Pre-award peer reviews for noncompetitive procurements valued at 
less than $500 million; and 

(3) Post-award peer reviews of all contracts for services valued at less than 
$1 billion. 

The DFARS mentions peer reviews in Subpart 215.270, Solicitation and Receipt 
of Proposals and Information; Subpart 207.104, General Procedures of Acquisition 
Planning; and Subpart 237.102, Policy of Service Contracts, but only to the extent of 
referring back to Subpart 201.170. The FAR, in and of itself, makes no reference to peer 
reviews. Since peer reviews are a DOD requirement, there is no regulatory requirement 
for the policy on peer reviews to appear in the FAR. 

3. Overview of Contract Attorney’s Desk Book Peer-Review 
Requirements 

The 2013 Contract Attorney’s Desk Book ( CADB ), published by the Contract and 
Fiscal Law Department of the DOD’s Judge Advocate General’s School, highlights the 
need for peer reviews. Chapter 8, Negotiated Procurements and Source Selection, 
mentions fairly early that peer reviews are a planning consideration to be considered in 
the acquisition planning phase, and reiterates the requirements stated in DFARS 201.170. 
The CADB does not provide any more details than what is included in the DFARS—only 
that it is a statutory requirement {CADB, 2013). The next section examines the 
procedures, guidance, and information (PGI) relating to the peer-review requirement. 

D. PROCEDURES, GUIDANCE, AND INFORMATION 201.170 
CONDUCTING PEER REVIEWS 

The DFARS (2014) contains implementation and DOD-specific supplementation 
information to the FAR (2014). The DFARS sets forth requirements established by law, 
policies that are implemented across the DOD, delegation authority, DOD-specific FAR 
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deviations, and other policies deemed to have a significant impact on the public. The PGI 
series offers supplementation to the DFARS. The PGI contains techniques, procedures, 
and guidance of a non-regulatory nature that are not included in the DFARS (DPAP, 
2013b). 

PGI 201.170, Conducting Peer Reviews , provides guidance on how to conduct 
peer reviews in accordance with DFARS 201.170. Since the PGI 201.170 is issued as 
guidance, rather than strict regulation, it provides a view into what the DPAP considers 
most important within the peer-review process. Consider the longest paragraph in the 
section: 


The results and recommendations that are products of peer reviews are 
intended to be advisory in nature; however, in the event the peer review 
report includes a recommendation that is identified as “significant” and the 
contracting officer does not intend to follow that recommendation, the 
senior procurement official of the contracting activity for the reviewed 
organization must be made aware of this fact before action is taken (or 
inaction, as applicable) that is contrary to the recommendation. Reviews 
will be conducted in a manner that preserves the authority, judgment, and 
discretion of the contracting officer and the senior officials of the 
acquiring activity. (DPAP, 2013b) 

This section would seem to indicate the importance placed on the peer-review 
results. It also highlights an interesting dichotomy between the sanctity of the contracting 
officer and the recommendations of the peer-review board. It is explicit in this statement 
that the contracting officer can stand by his or her decision, though it will be under the 
spotlight. 

PGI 201.170 begins by stating the review criteria that are the tenets of DOD-wide 
contractual actions, and includes a document highlighting these areas. This is the core of 
the peer-review process and the benchmark against which peer reviews are conducted. 
Figures 4 and 5 describe what participants in the peer-review process are examining for 
pre- and post-award acquisition of services. 
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Dol) Criteria for the Acquisition of Services (Preaward) 



Figure 4. DOD Peer-Review Criteria for Services Acquisitions during the 

Pre-award Phase (from DPAP, 2013c) 


18 


















DoD Criteria for tlie Acquisition of Services (Postaward) 


C </i 

c v> 
Ql < 


cc 


!-S 8 


"2 « 

^ J'D'C 

o-S 

tn .= in 
■ 5 - 2 a) O 

e _ _ 

E»-6c 

2^ » s 

CD < 1 > O 
O O ^ 

x O a> 
CL a> o Q. 


c « 
a> "O 

s g 

g is" 


“81 
c a> 
a> 

CLg 

8>g 
<D E 
■o 

§£ 

g* 

o ~o 
■2 c 
CO CO 

s-3 

{*■§ 

S"6 

£ C/> 

<D To 

CO o 

o 

~ o 

C 

a> t3 

i o- . 
S 

cn _ to 

<« s f= 
<■5 « 
a> 

0 

c _ 

S 8 


ilf co 


g to 


_ o 

i 2 

<U CO — 

CL ” ^ 

^ E w 

o « ^ 

O CD TO 

co o 1/1 

i- CO 

Cl <d 

o • g 
u £ E 


cr 



lit - 

§ &1-1 
CD >•* re 
03 ° T3 


g s 

E_ . 
0)0 w 


U TO 


"111! 


TL 


=» -D 


£ -- c <-> — :£ c 

I £ go $ o 5 

5 2 E« o <d w 

— Q Q- c a, 

^ O O _c 

TO Q -*£ % C 

- TO g O 
— T3 
aj c 

^S. 

r- a> 

o to ^ 

— - "tn o <*> 

1*1 HI 

c O £ o g 
5 = o £ ™ 

II g-£ 

| 2 O 5 
£ 2 
a> **=• o = 
22n2 

= £ 15 
g o 


u 

2 

S » 2 

u ^ 

a, <x> * 

io £ 

3 . 

CO a> 

if s 


& 


:«"g w. 

8-S 8 

£ -o 

ro 3 C 

a> T 5 re 
g£"2 

S « 2 

</> -Q 


A- S C “ * P» 

peSfilgS 

TO ■£ -£ > ajO o E 
Q. ‘ °- 


<u a> "g 8- 
"=-£*£ ~ = 
- S'! 8 -Si 

^ S CD— V> = 
^ -n n> 

• • m C (A _ _ 


>.2 «r 

0)5 | i»l 


.ti 


_ a> 
E Q 


^ O CL 

“* o 


co jo 


se $ « 

^ -O 


*31 

a> o 

co <J 


E 

o 

u 


a> O 


O) a> i2 
c -= — 


CO 


: cd 


.is 

3J « n 


2- 


'E ® 

a> > 

. _ - ° 
’5 o cr £ 

S = 2 


c 

S-:-i 
IT 0-0 

00 ^ 
e ™oS 

— ri — O 

0 ° -n 

■s 5.®^ 

•l: -o a) q: 
09 al< 


to co 



Figure 5. DOD Peer-Review Criteria for Services Acquisitions during the 

Post-award Phase (from DPAP, 2013c) 
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The objectives of the peer-review process are simple: to ensure that DOD 
contracting officers are implementing policy and regulations appropriately, to improve 
contracting as a process throughout the DOD, and to facilitate the sharing of ideas 
and best practices throughout the DOD. Additionally, the DPAP maintains a 
database of peer-review results for contracts reviewed at the DOD level, located at 
www.acq.osd.mil/dpap/cpic/cp/peer reviews.html , which is the subject of further 
analysis in this thesis (DPAP, 2013a). 

According to PGI 201.170, pre-award peer-review requirements are separated into 
two categories: competitive and noncompetitive acquisitions. Pre-award peer reviews for 
competitive acquisitions are required at three different points: issuing the solicitation, 
requesting the final proposal revisions, and issuing the contract award. For 
noncompetitive acquisitions, the requirement is for peer reviews at negotiation and 
contract award. For both competitive and non-competitive, post-award peer reviews 
concentrate on adequacy of competition, an assessment of contractor performance, and 
how adequate the government surveillance plan of contractor performance has proven to 
be (DPAP, 2013b). 

PGI 201.170 also defines the peer-review team: 

A senior official designated by the [Office of the Secretary of Defense] 

OSD Office of Small Business Programs will participate as a team 
member on peer reviews of services acquisitions. Teams will include 
civilian employees or military personnel external to the department, 
agency, or component that is the subject of the peer review. (DPAP, 

2013b) 

This PGI highlights the documents that peer-review teams must be able to access. 
Interestingly, there are also sections in the PGI that read like best practices for both the 
pre- and post-award phases: 

Pre-award Elements to be Addressed 

• The process was well understood by both government and Industry; 

• Source Selection was carried out in accordance with the Source Selection 
Plan and RFP; 

• The Source Selection Evaluation Board (SSEB) evaluation was clearly 
documented; 
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• The Source Selection Advisory Council (SSAC) advisory panel 
recommendation was clearly documented; 

• The Source Selection Authority (SSA) decision was clearly derived from 
the conduct of the source selection process; 

• All source selection documentation is consistent with the Section M 
evaluation criteria; and 

• The business arrangement. 

Post-award Elements to be Addressed 

• Contract performance in terms of cost, schedule, and requirements; 

• Use of contracting mechanisms, including the use of competition, the 
contract structure and type, the definition of contract requirements, cost or 
pricing methods, the award and negotiation of task orders, and 
management and oversight mechanisms; 

• Contractor’s use, management, and oversight of subcontractors; 

• Staffing of contract management and oversight functions; 

• Extent of any pass-throughs, and excessive pass-through charges by the 
contractor (as defined in section 852 of the National Defense 
Authorization Act for Fiscal Year 2007, Public Law 109-364); and 

• Steps taken to mitigate the risk that, as implemented and administered, 
non-personal services contracts may become de facto personal services 
contracts. 

The following section discusses how corporations review procurement actions 
versus how the DOD examines its procurement actions in the form of the peer-review 
process for contractual actions. 

E. CORPORATE PROCUREMENT MANAGEMENT, FEDERAL 

GOVERNMENT CONTRACTING, AND DOD CONTRACTING: A BRIEF 

COMPARISON 

Initially, a quick comparison of procurement in the corporate world against 
contracting within the federal government might lead the uninitiated to think they were 
similar. After all, both follow the same six-step process: procurement planning, 
solicitation planning, solicitation, source selection, contract administration, and contract 
closeout or termination (Garrett, 2007 p. 21). Under more detailed inspection, vast 
differences within the motivation of the parties involved and the forces that shape the 
process are exposed. Kovacs (2004) noted in his work Enhancing Procurement Practices , 
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The basic procurement principles and techniques are equally applicable in 
both the public and private sectors . . . which means: with the right quality, 
functionality and performance; under the right terms and conditions, 
among which costs and timely implementation usually are of essence; and, 
with the right long-term operation and/or maintenance support, (p. 175) 

The divergence begins with money in the form of funding and the method by 
which the government sector and private sector come by the funding. 

Corporations gain funding by financing debt or by spending their own profit they 
have retained in the corporation. In either instance, they are motivated by the need and 
desire to utilize those funds as efficiently as possible to earn the maximum profit possible 
from each dollar spent. Thus, there is no need for a regulatory requirement within the 
overarching corporate world for a formal peer-review process of contracts. As long as 
free market forces are in play, each corporation is driven by competition to be as efficient 
as possible (Kovacs, 2004). 

The chasm widens when considering business relationships and how they 
integrate with procurement in the corporate arena versus contracting in the government 
sector. Procurement actions outside of the government are typically more focused on the 
business relationship and building and maintaining a partnership for future ventures. 
These are the incentives, intangible and tangible, such as more favorable credit terms on 
purchases, that drive corporations to follow through on the whole procurement process. 
Hence, they are less apt to neglect the contract administration and contract closeout/ 
termination phases of the procurement process over federal contracting activities. 

Corporations do have methods for evaluating, or perhaps more accurately, 
examining how they are performing. Consulting services are often hired from sources 
outside of the corporation (or in cases of extremely large corporations, they may be 
contained in-house under a cloistered division). Consultants execute functions similar to 
what a peer review accomplishes, in that someone with an equal or greater amount of 
knowledge on the topic examines a subject (Garrett, 2007). The difference lies in the fact 
that, with the exception of extremely large contracts, consultant actions typically focus on 
a process, with an expected end state of further optimizing that process. Peer reviews of 
DOD contracts are conducted once Pre Award and once Post Award, as such the peer- 
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review process is evaluating an end product rather than the process that delivered the 
product. 

There is a tool that is designed to help both buyers and sellers evaluate risk and 
reward associated with procurements. “The Contract Management Risk and Opportunity 
Assessments Tool (CMROAT) is designed to help organizations, both buyers and sellers, 
assess the risk and opportunities associated with a pending/potential or actual contract'’ 
(Garrett, 2007, p. 234). The CMROAT has developed into a tool utilized by both 
government and private entities; however, it focuses on the risks and opportunities 
associated with the business dealings of the contract, and has less to do with the structure 
and content of the contract itself. 

Overall, the federal contracting process is process-driven; if the process is 
followed correctly, then the end product is typically in the acceptable range. 
Alternatively, as noted, the corporate procurement system is fueled by incentives. As a 
result, corporate review mechanisms tend to focus on optimizing a process for a greater 
return on investment, and the peer-review process as it applies to DOD contractual 
actions focuses on fixing an output of a process to bring it in line with what is deemed 
acceptable. 

F. SUMMARY 

In the literature review, I mentioned several aspects of the acquisition system and 
how they contributed to the peer-review requirement. With the implementation of the 
DAWIA, the DOD made significant strides in developing a professional acquisition 
workforce. Negligence of the workforce throughout the late 1990s and into the 2000s led 
to a shortage within the acquisition workforce, followed by a surge in acquisition activity 
accompanying the rapid increase in overseas contingency operations. With the 
contracting system thoroughly stressed, numerous examples of inefficiencies, fraud, 
waste, and abuse were uncovered. This led to an official peer-review requirement from 
Congress. The actual peer-review process was developed based on an existing pilot 
review program, and expanded to meet the congressional requirement. The following 
chapter discusses the research methodology. 
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III. METHODOLOGY 


A. INTRODUCTION 

This chapter addresses how I collected the data and used the analytics. It also 
includes a summary of the DPAP database of peer-review results to provide context for 
the data analysis chapter of my research. The various aspects of the DPAP database of 
peer-review results are also described. The purpose of my research was not to provide an 
exhaustive answer to an in-depth research question; rather, the purpose was to explore a 
previously uncharted research area to examine if it merited further study. The DPAP 
database of peer-review results was broken down categorically by the various data 
elements of data present and analyzed quantitatively based on frequency of occurrences 
of a particular data element. 

B. DPAP DATABASE OF PEER-REVIEW RESULTS OVERVIEW 

The DPAP database of peer-review results resides in the contract policy section of 
the DPAP website, in both PDF and Excel versions. The purpose of the DPAP database 
of peer-review results is to collect the results from peer reviews of contracts that met the 
threshold ($1 billion and above) for DOD-level peer review. According to the Peer 
Review section of the DPAP website: 

The Peer Review program improves the quality of the Department’s 
contracting processes by sharing lessons learned and best practices and 
ensuring that contracting officers implement policy and regulations in a 
consistent manner. The program is implemented in accordance with DODI 
5000.02 and DFARS Part 201. (DPAP, 2014) 

1. Summary of the Parts of the DPAP Database of Peer-Review Results 

The DPAP database of peer-review results itself is organized in a simple format. 
Figure 6 shows an example page from the DPAP database of peer-review results as of 
December 20, 2013. 
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OSO Level Peer Reviews 

Best Practices. Lessons Learned, and Recommendations (as of December 20. 2013) 



Figure 6. Excerpt from DPAP Database for OSD-Level Peer Reviews 

(from DPAP, 2013a) 
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Each row consists of an entry for a particular data element of a contract, and 
the columns contain the appropriate type of data related to that contract entry. The 
DPAP database of peer-review results has been anonymized, so there is no way to 
discern how many entries apply to the same contract. The Category column lists the 
category the peer-review comment falls within. There are eight Categories: Incentive and 
Award Fee, Market Research, Peer Review, Post-award Administration, Pricing, 
Requirements/Performance Work Statement/Scope of Work, Source Selection, and 
Terms and Conditions. The Types of Contracts column spans a staggering 31 different 
types, consisting of competitive and non-competitive, services, goods, and construction 
contracts. 

The Feedback column provides a simple narrative comment regarding the opinion 
of the peer-review team. The column entitled Review Phase illustrates the phase of the 
contracting process. There are currently six phases: Phase 1, Phase 2, Phase 2&3, Phase 
3, Phase 4, and Post Award 1. The final column relates to the Type of Feedback provided, 
and consists of three categories: Recommendation, Best Practice, and Fessons Fearned. 
Figure 6 summarizes the different database fields. 

2. Phases of the Contracting Process versus DPAP Peer-Review Phases 

The DPAP database of peer-review results provides no explanation about the 
timing of the Review Phases used, nor does the Frequently Asked Questions or Standard 
Operating Procedures for peer reviews on DPAP’s website. Further research uncovered 
an explanation of four phases in the GAO (2010) report Status ofDOD's Implementation 
of Independent Management Reviews for Sendees Acquisitions. Phase 1 is prior to the 
issuance of the solicitation. The documents reviewed consist of the performance work 
statement, quality assurance surveillance plan, request for proposal, and source selection 
plan. Phase 2 is prior to the request for final proposal revisions. The documents reviewed 
are instructions for proposals and proposal evaluation criteria, source selection evaluation 
guide, source selection plan, and evaluations of contractor proposals. Phase 3 is prior to 
contract award. The documents reviewed consist of the proposal-analysis report and 
selection-decision document. Phase 4 is the post-award review. The documents reviewed 
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consist of any documentation related to the program, such as task orders, award-fee plan, 
and performance assessments (GAO, 2010, p. 6). Figure 7 highlights the comparison 
between the DPAP Peer Review Phases and phases of the contracting process. 


DPAP Peer 

ReviewPhases 

1) Prior to issuance of the solicitation 

(No comparable 
step) 

2) Prior to request for final proposal 
revisions & 3) Prior to contract award 

4) Post-award review 

Contracting 
Process Phases 

1) Procurement 
Planning 

2) Solicitation 
Planning 

3) Solicitation 

4) Source Selection 

5) Contract 
Administration 

6) Contract 
Closeout 


Figure 7. DPAP Peer-Review Phases Compared to the 
Contracting Process Phases 


The DPAP database of peer-reviews Review Phases actually contain Phases 1 
through 4, and an additional Post-award 1 phase. Analyzing the Narrative feedback seems 
to indicate that in practice, the Phase 4 review described in the aforementioned GAO 
(2010) report occurs shortly after contract award, and the Post-award 1 review occurs 
well after contractor performance is underway. 

C. TYPES OF DATA ANALYSIS 

1. DPAP Database of Peer-Review Results Data Element Fields 

Within the DPAP database of peer-review results, there are 288 records. Each 
data element was subdivided within the spreadsheet to allow for the identification of the 
frequency of occurrence for each data element. This data was then used to identify trends 
within the DPAP database of peer-review results. Descriptive analysis of the frequency of 
observations for each category of feedback was conducted. 

The eight Categories, 31 Types of Contracts, six Review Phases, and three Types 
of Feedback were individually extracted from the DPAP database of peer-review results. 
The results of each data element was tallied and recorded and then analyzed by frequency 
of occurrence against the other data elements, with the goal of identifying overarching 
trends in the data elements. 

Since the preponderance of the data elements were contained in Recommendation 
Type of Feedback, the Category, Type of Contract, and Review Phase data elements were 
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further analyzed with an eye toward identifying how much Recommendation Type of 
Feedback appeared in each of those data elements. To provide additional granularity, 
occurrences of Recommendations were expressed as percent of the total occurrences for 
both Categories and Contract Type data elements. 

Competency gaps identified in the DOD Contracting Workforce Competency 
Assessment Final Report (DPAP, 2007) were cross-referenced against the DPAP 
database of peer-review results to identify any areas of overlap between the two. Once 
areas of overlap had been identified, occurrences of Recommendation Type of Feedback 
for the overlap areas was reviewed to see if the identified competency gaps had increased 
occurrences of Recommendations. 

2. Narrative Feedback Themes 

The Feedback section for each entry was also examined to identify any common 
themes within the feedback provided. The primary goal for examining the content of the 
Feedback section was to identify apparent incongruities between listed categories, such as 
feedback type, and the actual narrative posted within the Feedback column. My 
examination is concerned with identifying irregularities with policy and regulations. 

D. SUMMARY 

The Methodology chapter provided a brief overview of the DPAP database of 
peer-review results, and how it was subdivide for analysis. The Review Phases were 
compared to the phases of the contracting process. How the narrative Feedback themes 
were reviewed was also addressed. The next chapter, Chapter IV, presents the results of 
the data analysis. 
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IV. DATA ANALYSIS 


A. INTRODUCTION 

The previous chapter discussed how the data was analyzed. This section focuses 
on trends within the DPAP database of peer-review results by various data elements of 
the data. Frequencies of occurrences by Category, Type of Contract, Review Phase, and 
Type of Feedback are examined. The Recommendation Type of Feedback receives 
additional review because of its predominance of occurrences within the DPAP database 
of peer-review results. Relationships between the DOD Contracting Competency Model 
(CCM) and trends within the DPAP database of peer-review results are compared. 
Finally, themes within the narrative feedback section are also investigated. 

B. DESCRIPTIVE STATISTICS 

The various data elements within the DPAP database of peer-review results noted 
in the methodology chapter were analyzed for frequency of occurrence. The importance 
of breaking down the DPAP database of peer-review results in this manner lies in the fact 
that it allows contracting officers and policymakers to identify areas where systemic 
issues within the contracting process persist. The data on frequencies of occurrences and 
the underlying trends can then be used to target the development of educational and 
training programs to address problem areas. A process as complex as developing high- 
dollar defense contracts may well never be without faults, but previous deficiencies need 
to be addressed to avoid future duplication of effort. 

1. Category 

Working across the DPAP database of peer-review results, the first data element 
encountered is the Category column. The largest category was the source selection 
category at 115 of 288 observations, which should not be surprising given two facts: 
(1) the emphasis that is placed on the source selection process within DOD contracting; 
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and (2) the number of protests that originate in the source selection process, 
stemming from allegations of issues such as unequal application of evaluation criteria, 
unfair negotiation and discussion practices, and poor use of Best Value trade-offs and 
Lowest Price Technically Acceptable criteria for contract award (White, 2012). Figure 8 
represents the occurrences of all the category types represented in the DPAP database of 
peer-review results. 


Terms & Conditions 

Source Selection 
Requirements/PWS/SOW 
Pricing 
Post Award Admin 

Peer Review 

Market Research 

Incentive and Award Fee 


Occurrences by Category Type 
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Figure 8. Frequency Distribution of Category Type (after DPAP, 2013a) 
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The second cluster of occurrences centered on Terms and Conditions, Pricing, and 
Post Award Administration. It is notable that pricing, while the second largest category, 
is still a distant second with 46 occurrences. Given issues that seem to pervade the pricing 
realm within contracting, it would seem that this area would receive more attention. 
Bringing the point into sharper focus is the consideration that this data represents high- 
dollar contracts valued at $1 billion and above, and that high value defense procurements 
often have significant cost overruns exceeding 20% of the original cost estimate (GAO, 
2014). 


2. Type of Contract 

The next data element of the DPAP database of peer-review results is the Type of 
Contract column. The contract types listed comprise a staggering 31 types of contracts. 
The most frequently occurring type is the Competitive Multiple Award Services Contract 
at 59 occurrences, followed by the Competitive Services Contracts and Competitive 
Multiple Award indefinite delivery indefinite quantity (IDIQ) at 39 and 20 occurrences, 
respectively. Figure 9 shows the various types of contracts listed and the number of times 
the occurrence was observed. 
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Occurrences by Contract Type 
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Weapon System Development 
Sole Source Procurement of Services 
Non Competitive Weapon System Development 
Non Competitive Weapon System 
Non Competitive Supplies Contract 
Non Competitive Services Contract 
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Figure 9. Frequency Distribution of Contract Type (after DPAP, 2013a) 
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Admittedly, the terminology of some of the listed contract types could be 
combined, such as Competitive Supplies Contract and Competitive Procurement of 
Supplies. The issues associated with having such a broad scope of contract types listed is 
addressed in a subsequent section of this analysis because this particular part of the 
DPAP database of peer-review results is one of two that could benefit the most from 
refinement. The key takeaway from this section of data appears to be that, overall, the 
DOD is peer reviewing more contracts for services than anything else. Service contracts 
comprise over 65%, or 189 occurrences, of the Contract Type entries. This is not 
surprising, considering the amount of services contracted for within the DOD, but what 
is worth considering is that those 189 occurrences represent a dollar value of at least 
$1 billion each (GAO, 2008). 

3. Review Phase 

The next data element to address is the Review Phase column. Again, one area 
contains the majority of the occurrences; this time they fall into Phase 1, with a total of 
157. Phase 2 follows with 64, and Phase 3 and Post-award 1 are nearly tied at 28 and 
27 occurrences. Figure 10 shows the occurrences by review phase. 


Occurrences by Review Phase 



i Occurrences 


Figure 10. Frequency Distribution of Review Phase (after DPAP, 2013a) 
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The trends demonstrated within the occurrences by review phase would seem to 
indicate that learning is taking place as the contract process moves along. It also seems to 
break itself naturally between Phase 4 and Post-award 1, when the contract would have 
been awarded at the conclusion of the source selection process and contract performance 
had begun. This action would move the contract into the contract administration portion 
of the contracting process. Since contract administration represents a very different phase 
of the contracting process from procurement planning, solicitation planning, solicitation, 
and source selection, it follows that additional learning would begin to occur in the Post¬ 
award 1 phase. 

4. Type of Feedback 

The final quantifiable data element of the DPAP database of peer-review results is 
the Type of Feedback column. This column seeks to assign a type to the narrative 
feedback column. Consistent with the other DPAP database of peer-review results 
categories, one category heavily outweighed the others with occurrences. 
Recommendation contained 190 occurrences, with Best Practice containing 68 and 
Lessons Learned amassing 30 occurrences. Figure 11 demonstrates the occurrences by 
Type of Feedback. 


Occurrences by Feedback Type 
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Figure 11. Frequency Distribution of Type of Feedback (after DPAP, 2013a) 
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The type of feedback aspect is the other data element of the DPAP database of 
peer-review results that could benefit from additional refinement. The Lessons Learned 
Type of Feedback ostensibly provides a way to communicate an item of information the 
peer-review team was not aware of to the readership of the DPAP database of peer- 
review results. In reading the narrative, it often becomes difficult to distinguish between a 
Lesson Learned and a Recommendation. Best Practices are just that—a best practice that 
the peer-review team observed from the organization or contract that the team was 
reviewing; it exists as a way to propagate those practices back out to the contracting 
workforce. With only three types of feedback categories, and since most of the 
occurrences are in a single category, it is difficult to ascertain anything quantitatively 
unless the recommendation category is delved into further, which occurs in the next 
section. 


5. Recommendation Category 

The Recommendation category contained 190 out of 288 occurrences regarding 
feedback type. That comprises 65.97% of the total feedback types. It is also the single 
largest density of occurrences anywhere in the DPAP database of peer-review results. As 
such, further analysis was conducted. 

Examining the phases revealed that most Recommendations were concentrated in 
Phase 1 reviews. Phase 1 contained 130 Recommendation occurrences. Phase 2 was a 
distant second at 38 occurrences, with 12 occurrences in Phase 3, none in Phase 4, and 11 
in Post-award 1. Figure 12 elucidates the recommendation occurrences by phase. 
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Recommendation Occurrences By Phase 
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Figure 12. Frequency Distribution of Recommendations by Phase 

(after DPAP, 2013a) 

The findings are consistent with the idea that learning is taking place throughout 
the contracting process, and organizations that are subject to formal review by a peer- 
review team are applying the observations of the peer-review team. It again demonstrates 
the break between contract award and contract performance, illustrated by the uptick 
in Recommendations between the Phase 4 Review Phase and the Post-award 1 Review 
Phase. 

Next, the number of occurrences of recommendations within each Category was 
examined. The categories were Terms & Conditions, Source Selection, Requirements/ 
PWS/SOW, Pricing, Post Award Admin, and Incentive and Award Fee. Figure 13 
exhibits the number of occurrences of Recommendations within each Category. 
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Occurrences of Recommendation 
Within Categories 
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Figure 13. Frequency Distribution of Recommendations within Categories 

(after DPAP, 2013a) 


Another important way to view this data is by percentage of Recommendations 
within each Category. 


Categories 

Total 

Occurrences 

Occurrences of 
Recommendation 

Percent of 
Recommendation 

Incentive and Award 
Fee 

23 

20 

86.96 % 

Market Research 

4 

0 

0% 

Peer Review 

1 

0 

0% 

Post-award Admin 

35 

9 

25.71 % 

Pricing 

46 

41 

89.13 % 

Requirements/PWS/ 

SOW 

24 

17 

70.83 % 

Source Selection 

115 

66 

57.39 % 

Terms & Conditions 

40 

37 

92.5 % 

Total 

288 

190 

65.97 % 


Table 1. Percentage of Recommendations by Category 
(after DPAP, 2013a) 
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Table 1 lists each of the Category types in the DPAP database of peer-review 
results. Total Occurrences is the number of times that Category appears within the 
database. Assuming that the priority of the DPAP database of peer-review results 
is to capture issues with solicitations and contracts to assist in informing the DOD 
contracting workforce, then the third column of the above chart is the most important 
cross section of the data to consider. It details the number of Recommendations per 
Category within the DPAP database of peer-review results. Consider the first row of 
Table 1; it expounds on the Incentive and Award Fee Category. That Category appeared 
23 times in the DPAP database of peer-review results; of its 23 appearances, 20 of those 
constituted Recommendation Feedback Type, with 86.96% of the occurrences being 
Recommendations. The percentage column is included simply to help visualize the 
Recommendation occurrences by category. This provides a more precise picture within 
the DPAP database of peer-review results as to where peer-review teams are uncovering 
issues. If one only examined the occurrences of Recommendations, the conclusion might 
be that 20 Recommendations might not be significant, but once the dimension is added 
that 20 is out of 23, and 86.96% of the Feedback Type for that Category is a 
Recommendation, then there may be an issue within Incentive and Award Fees that needs 
to be addressed. By scrutinizing the data in that manner, it would seem the areas of 
concern that should merit additional attention are Incentive and Award Fee, Pricing, 
Requirements/PWS/SOW, and Terms & Conditions. Curiously enough, while Source 
Selection as a Category is the most frequently occurring Category within the DPAP 
database of peer-review results, only 57.39% of its appearances consist of 
Recommendations, with the rest comprised by Best Practices and Lessons Learned. 

The final portion of the DPAP database of peer-review results to be examined in 
relation to Recommendations is by Contract Type. Figure 14 lists the number of 
occurrences of Recommendations within each listed Contract Type. 
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Occurrences of Recommendation Type Feedback by 
Contract Type 
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Figure 14. Frequency Distribution of Recommendations by Contract Type 

(after DPAP, 2013a) 
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Similarly to the way the data for recommendations within each category was 
dissected. Table 2 provides a breakdown of percentage of each contract type that returned 
a recommendation. 


Contract Type 

Total 

Occurrences of 
Recommendation 

Percentage of 
Recommendation 

Commodity (Competitive) 

4 

3 

75 % 

Competitive Design/Build FPIF 

2 

2 

100% 

Competitive IDIQ 

1 

1 

100% 

Competitive IDIQ for IT Products 

4 

4 

100% 

Competitive IDIQ Services Contract 

6 

2 

33.33 % 

Competitive IQC 

1 

1 

100% 

Competitive Multiple Award (Combination 

7 

7 

100% 

Multiple Award FFP and Cost type line items) 




Competitive Multiple Award IDIQ R&D 

1 

0 

0% 

Contract 




Competitive Multiple Award IDIQ Services 

22 

3 

13.64% 

Competitive Multiple Award Services 

59 

41 

69.49 % 

Contract 




Competitive Multiple Award IDIQ 

5 

2 

40% 

Competitive Multiple Award IDIQ Supplies 

2 

1 

50% 

Contract 




Competitive Procurement of Services 

20 

9 

45 % 

Competitive Procurement of Supplies 

14 

9 

64.29 % 

Competitive Services Contract 

39 

16 

41.03% 

Competitive Supplies Contract 

1 

1 

100% 

Competitive Weapons System Development 

16 

14 

87.5 % 

Design/Build Construction Contract 

3 

3 

100% 

Logistical Services (Competitive) 

10 

7 

70% 

Multiple Award Construction Contract 

2 

2 

100% 

Multiple Award Contract for R&D 

6 

5 

83.33 % 

Non-Competitive Multiple Award Services 

11 

11 

100% 

Contract 




Non-Competitive Services Contract 

4 

4 

100% 

Non-Competitive Supplies Contract 

10 

8 

80% 

Non-Competitive Weapon System 

8 

8 

100% 

Non-Competitive Weapon System 

6 

5 

83.33 % 

Development 




Sole-Source Procurement of Services 

1 

1 

100% 

Weapon System Development 

5 

4 

80% 

Weapon System Development Sole Source 

1 

0 

0% 

Weapon System Production Lot Buy (Sole 

3 

3 

100% 

Source) 




Weapon System Technology Demonstration 

14 

13 

92.86 % 

Total 

288 

190 

65.97 % 


Table 2. Percentage of Recommendations by Contract Type (after 

DPAP, 2013a) 
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The variety of contract types listed makes it difficult to pinpoint precise issues 
within the Contract Type data element. If we aggregate the Service type contracts, then 
the rate of Recommendations is 49.47%. This does not represent an overwhelming rate 
given the concentration of Recommendation Type Feedback that some data elements in 
the DPAP database of peer-review results have experienced. Scanning Table 2 would 
seem to indicate that the Contract Type data element is garnering more than its fair share 
of Recommendations; indeed, rates upward of 70% seem prevalent. Two Contract Type 
data elements are drastically pulling the average down. Competitive Multiple Award 
IDIQ Services with three Recommendations in 22 occurrences, and Competitive 
Procurement of Services with nine Recommendations in 20 occurrences. With the wide 
range of Contract Types employed in this data element, it will take more entries in the 
DPAP database of peer-review results to draw meaningful conclusions about where 
issues with Contract Type exist. The next section will address comparisons between the 
Contracting Core Competencies and the DPAP database of peer-review results. 

C. CONTRACTING CORE COMPETENCIES COMPARISONS 

The DOD CCM was developed in 2007 after an exhaustive survey was conducted 
across multiple DOD contracting organizations. The survey asked both employees and 
supervisors to assess critical areas within the contracting process they worked on, and 
also asked them to rate themselves from a proficiency standpoint. Subject matter experts 
from the contracting career field took a list of 27 business and professional competencies 
and were asked to rank them as to how effective they were in relation to job performance. 
Ten competencies were identified as the most important for work performance. From 
that, the CCM was developed, consisting of 12 units of competence, 28 technical 
competencies, and 10 professional competencies (DPAP, 2007). Table 3 represents the 
CCM. 
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DoD Contracting Competency Model 
established March 2007 


Pre-Award and Award 

Determmaoon of How Best To Satisfy Requirements 

Considers Socio-economic Requirements 

Promote Competition 

Source Selection Piannrg 

SotctaDon ofOfers 

Responsfekty Deternwiaocn 

3d Evaluation (Sealed Bidding) 

Proposal Evaluation (Contraong by Negotiation) 

Source Selection 

Contract Award 

Process Protests 

Develop and/or Negotiate Posaons 

Jusifcaoon of Other than Ful and Open Compewon 

Terms and Conditions 

Preparation and Negotiation 

Advanced Cost and/or Pnoe Analysis 

Advanced Cost and/or Price Analysts 

Contract Admmstrauon 

Irmaaon of work 

Contract Performance Management 

Issue Changes and Mediations 

Approve Payment Requests 

Close-out Contract 

Smal Busness/Sooo-Economic Programs 

Addressing Smal Busness Concerns 

Negotiate Forward Pnang Rate Agreements 
and Administer Cost Accoumnq Standards 

Negotiate Forward Pnang Rate Agreements 
and Administer Cost Accoumnq Standards 

Contract Termmaoon 

Contract TermriaDon 

Procurement Pokey 

Procurement .Analysis 

Other Competencies 

E-Business and Automated Tools 

Acthrty Program Coordinator for Purchase Card 
Construcson/Archiect and Engmeenng 

Contracting in a Contingent and/or Combat Environment 

Contracting n a Contingent and/or Combat Environment 

Professional Competency 

Problem Sohnng 

Customer Service 

Oral Communication 

Written Communication 

Interpersonal Skis 

Decisiveness 

Technical CredMty 

FlexMy 

Resaence 

Accountably 


Table 3. DOD Contracting Competency Model (from DPAP, 2007) 
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The final report detailing the survey results and providing the model was entitled 
Department of Defense Contracting Workforce Competency Assessment Final Report 
(DPAP, 2007). Included in the report were high-risk areas within the contracting 
workforce that had been identified through the survey process. These high-risk areas 
were described as “Competency Gaps.” The following is a list of the areas established as 
Competency Gaps: 

• Fundamental contracting skills across entry and journey levels of the 
Contracting Workforce and currency, breadth, and depth of knowledge 
across journey and senior levels; 

• The Source Selection Process 

• Cost and Price Analysis 

• Contract Performance Management 

• Integrated Acquisition Skills. (DPAP, 2007) 

From these Competency Gaps, a few areas from the DPAP database of peer- 
review results can be cross-referenced. Within the DOD CCM, the source selection 
process is broken down into the following subparts: source selection planning, source 
selection, proposal evaluation, and contract award. Referencing the previous section on 
the Recommendation Feedback Type breakdown by phase, the greatest frequency of 
Recommendations by phase was in Phase 1. Phase 1 would correlate with source- 
selection planning within the CCM, but since the Department of Defense Contracting 
Workforce Competency Assessment Final Report (DPAP, 2007) did not break out any of 
the sub-areas within the source selection process as being any more at risk than any other 
area, it is difficult to ascertain whether there is a specific relationship between the report 
results and the observations from the DPAP database of peer-review results. The obvious 
spike in Recommendations in Phase 1 can be generalized as an issue with the overall 
source-selection process, which helps to corroborate the survey results on which the 
CCM was based. 

Two areas from the DPAP database of peer-review results that can be more 
directly related to the CCM are the Pricing and Incentive & Award Fee categories. Recall 
that of the three Types of Feedback, Recommendations comprised 89.13% of the 
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feedback for the Pricing category and 86.96% of the feedback for the Incentive & Award 
Fee category. These two categories garnered the highest densities of Recommendations 
within the DPAP database of peer-review results, providing an indication that these areas 
are presenting difficulties to the contracting workforce. This also shows a relationship to 
the Competency Gaps of cost and price analysis. Unfortunately, without having dates 
associated with the DPAP database of peer-review entries, it is difficult to ascertain 
whether the training initiatives that were driven by the Department of Defense 
Contracting Workforce Competency Assessment Final Report (DPAP, 2007) are having a 
noticeable impact on the number of Recommendations that these two categories are 
attracting from the peer-review teams. 

The final area identified as a Competency Gap that has a cross reference with the 
DPAP database of peer-review results is the number of Recommendations within the 
Post-award 1 Review Phase, albeit tentatively so. Contract performance management was 
identified as Competency Gap, which would align with the Post-award 1 Review Phase 
of the DPAP process. The number of Recommendations showed an uptick between Phase 
4, with no recommendations, to 11 Recommendations in the Post-award 1 Phase. The 
link remains tenuous, however, and more contracts will need to be reviewed in the Post¬ 
award 1 Phase to strengthen the tie between the peer-review results and the Competency 
Gap. The following section addresses themes within the narrative feedback data element 
of the DPAP database of peer-review results. 

D. NARRATIVE FEEDBACK THEMES 

The narrative feedback data element of the DPAP database of peer-review results 
provides an opportunity for the peer-review team to enter its comments on the occurrence 
documented. Although difficult to summarize or quantify the narratives, some themes did 
occur within the comments. 

The most startling, if not alarming, issue is the episodes that reveal direct 
violations of the FAR or DFARS. Concerns existed about improperly proposed award 
fees, award fees that were not necessary, and offerors being allowed to propose their own 
fee pool percentage. In another instance, a contracting officer attempted to provide a 
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performance incentive for a project in progress review that was already mandated by a 
DFARS clause. Proposals with vague requirements were also a recurring theme. 

Questions over training adequacies persisted as well. Some comments revolved 
around concerns over the use of contracting officer representatives and whether they had 
received proper training to execute their intended function. One training comment stood 
out above the others: “Minimal source selection experience within the government 
evaluation team has resulted in heavy reliance on contractor expertise. Recommend 
obtaining assistance from the DAU in developing the source selection framework” 
(DPAP, 2013a). 

The narrative comments read like a summary of GAO and Inspector General (IG) 
reports that have detailed the deficiencies with contracting over the last several years. The 
disturbing aspect is that these entries represent contracts valued at $1 billion and above. 
Presumably, with such a high dollar value, DOD agencies and departments would have 
experienced contracting officers and dedicated contracting teams developing these 
contracts. Yet mistakes that would be associated with an inexperienced contracting team 
are appearing in the feedback. The next section contains recommendations for 
improvements to the DPAP database of peer-review results. 

E. RECOMMENDATIONS FOR IMPROVEMENTS TO THE DPAP 

DATABASE OF PEER-REVIEW RESULTS 

1. Introduction 

This section addresses recommendations for improvements to the DPAP 
database of peer-review results based on the data analysis conducted. The following 
recommendations would improve the functionality and usefulness of the DPAP database 
of peer-review results. An additional category should be added to the DPAP database of 
peer-review results to provide what basic contract type is being utilized in the entry. The 
existing Contract Type data element should be repurposed as a Requirement Type data 
element. The Type of Feedback data element needs to add another category to address 
regulatory requirements. The Source Selection Category should be further subdivided to 
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address the phases that make up the source selection process. Lastly, labelling 
conventions across DPAP produced products should be consistent. 

2. Addition of a Data Element 

The first recommendation is to add another data element to the DPAP database of 
peer-review results. Currently, there is a Contract Type data element, but it does not 
simply list the commonly used types of cost and fixed-type contracts. The current 
Contract Type field should be relabeled as Contract Requirement Type to retain 
descriptive value and context for the entry being examined. The Contract Type field 
should be repurposed, containing one of the basic contract types. As the DPAP database 
of peer-review results stands, the state of the Contract Type field negatively impacts data 
analysis. This is due to the difficulty, if not down-right impossibility, of identifying 
which basic contract type is being employed in regard to the data entry. If basic contract 
types were listed, then basic analytics could be conducted to identify which, if any, 
contract types are garnering more negative feedback than others. 

3. Regulatory Requirements 

The second recommendation addresses the Recommendation feedback type. The 
Recommendations make up the majority of the DPAP database of peer-review results 
entries and address a myriad of issues, some small, some large. The issue that needs to be 
addressed is Recommendations that are regulatory in nature. For example, consider this 
entry “An Award Fee plan needs to conform with the FAR Rule 16.401(e) (DPAP, 
2013a).” This Recommendation targets a violation of a regulation, in this case FAR 
16.401(e); thus, this particular Recommendation carries far more weight than might 
normally be associated with a recommendation. To call attention to regulatory violations, 
the DPAP database of peer-review results needs to contain a “regulatory” Feedback Type 
that can be used to pinpoint such violations. It would provide another method for 
policymakers to determine over time which areas of the FAR are habitually 
misunderstood or underutilized so training can be tailored to address those shortcomings. 
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4. Subdivide the Source Selection Category 

The Source Selection Category would benefit from additional refinement, adding 
to the overall usefulness of the DPAP database of peer-review results. The Source 
Selection Category contained 115 entries out of the 288 total entries in the database. 
By comparison, Pricing came in a distant second at 46 entries. Source Selection as its 
being used currently within the DPAP database of peer-review results covers four distinct 
phases of the contracting process. Those phases are procurement planning, solicitation 
planning, solicitation, and source selection. The Source Selection category should be 
further subdivided into categories that mirror the aforementioned phases, again for the 
purposes of providing an accurate assessment of where problems are occurring within the 
contracting process and enabling policy makers to address deficiencies. 

5. Ensure Labeling Conventions Are Consistent 

The final recommendation addresses labeling conventions. Phases, titles, contract 
types, and so forth within the DPAP database of peer-review results do not consistently 
match up well with other contracting related products, some of which were created by the 
DPAP organization itself. For example, it is difficult to cross-reference any area of the 
DPAP peer-review database with the CCM. The CCM was developed by DPAP shortly 
before the peer-review requirement was established in 2008. Assuming the DOD is 
serious about improving the performance of its contracting workforce, then should not 
labels on the DPAP database of peer-review results allow a user to cross-reference 
competencies that have been identified as critical for accomplishing contracting 
functions? The subsequent section encompasses the summary of the data analysis 
chapter. 

F. SUMMARY 

This chapter discusses the results of the analysis of the DPAP peer reviews. 

Trends within the DPAP database of peer-review results were examined. Frequencies of 

occurrences by Category, Type of Contract, Review Phase and Type of Feedback were 

inspected. The Recommendation Type of Feedback received additional attention. The 

data showed the majority of feedback within the DPAP database of peer-review results 
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occurred in the Recommendation Feedback Type, and in the initial Review Phases. 
Further examination reveals that the majority of the occurrences of the Pricing, Incentive 
and Award Fee, & Requirements/PWS/SOW Categories consist of Recommendation 
Feedback Type. Applicability of relationships within the data to previously identified 
Competency Gaps within the DOD contracting workforce was examined. Themes were 
identified within the narrative feedback comments that showed issues requiring corrective 
action within high-dollar DOD contracts. The next chapter summarizes and concludes the 
research and provides areas for further research. 
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V. SUMMARY, CONCLUSIONS, AND AREAS 
FOR FURTHER RESEARCH 


A. SUMMARY 

Since the early 1990s, the number of personnel within the acquisition workforce 
has steadily declined. Since the onset of the overseas contingency operations in 2002, 
DOD contracting actions have risen astronomically. Additionally, since the reliance on 
contracts has increased, DOD shows no signs of decreasing that reliance for 
accomplishing its mission. With the reduction in the workforce and the rapid increase in 
the number of contract actions and the amount of dollars expended, issues within the 
contracting workforce have also risen dramatically, mostly as a result of a workforce that 
is either too inexperienced or spread too thin for its assigned task. DPAP implemented 
the independent management review (or peer review) requirement in 2008 as a means of 
attempting to compensate for the lack of experienced contracting personnel within its 
ranks. The next section examines the conclusions of this thesis. 

B. CONCLUSIONS 

Research Question 1: Are there trends within the peer-review results of 
DOD-level peer-reviewed contracts? 

Trends were identified in the DPAP database of peer-review results. Not 
surprisingly, most of the entries consisted of Recommendation Type Feedback, at 65% of 
the entries; however. Best Practice Feedback Type was represented at 24% and Lessons 
Learned at 11%. This demonstrates the DPAP database of peer-review results potential as 
a tool to spread effective contracting practices throughout DOD. The analysis also 
showed that the entries were largely Source Selection focused, 110 out of 288 entries 
pertained to Source Selection. The next most prevalent occurrence within Category Type 
was Pricing with 46 entries. Potential issues were also indicated with the following 
Categories: Terms & Conditions, Requirements/PWS/SOW, and Incentive & Award Fee. 
Those categories amassed 70% or greater Recommendation Type Feedback. 
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Research Question 2: Are any trends identified related to the competency 
gaps identified in the Department of Defense Contracting Workforce Competency 
Assessment Final Report ? 

In regard to trends between the DPAP database of peer-review results and 
competency gaps from the Department of Defense Contracting Workforce Competency 
Assessment Final Report (DPAP, 2007), the strongest link was between issues with Cost 
and Pricing Analysis identified in the Final Report and the DPAP database of peer- 
review results Pricing and Incentive & Award Fee Categories. Pricing consisted of 89% 
Recommendation Feedback Type, and Incentive & Award Fee consisted of 87%. 

This research identified trends within the results provided by the peer-review 
teams and recorded within the DPAP database of peer-review results. The results of the 
study can be used for further research into areas such as identifying recurring problem 
areas, identifying training gaps, and answering other “why” type questions associated 
with the results of this study. This study was by no means intended to be exhaustive in 
nature, but rather to investigate an area that had previously received little, if any, 
attention. 

The DPAP database of peer-review results is a good tool for the DOD contracting 
workforce; it does require some refinement to reach its full potential, as well as the 
implementation of a dedicated feedback mechanism to disseminate the peer-review 
results back to the contracting workforce. An important consideration that comes into 
focus is the fact that the trends show the same issues facing lower dollar threshold 
contracting actions are impacting high-dollar contracting actions as well. However, since 
the DPAP database of peer-review results threshold level is above $1 billion, the effects 
of mistakes in contracting are magnified. 

C. AREAS FOR FURTHER RESEARCH 

As DPAP’s database of peer-review results accumulates entries over time, the 
statistical population for analysis will increase. The current number of entries within the 
DPAP database of peer-review results allows for trend and frequency analysis within the 
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database itself. However, once the various fields are broken down into their constituent 
parts, more entries would allow for more meaningful comparisons. 

Specifically, there are two areas to consider. First, as Best Practice feedback types 
continue to accumulate, is there a relationship with the Contract Management Maturity 
Model (CMMM)? CMMM assessments contain assessments of process strength, process 
results, and management support. Do the narrative comments in the DPAP database of 
peer-review results best practice entries support any of those? Additionally, if service- 
level peer-review results could be obtained, a more direct comparison could be made with 
respective CMMM assessments from that service. Second, since the DPAP database of 
peer-review results is a relatively new construct, as more contracts enter the Post Award 
1 review phase, will we continue to see an uptick in recommendations (see Figure 11)? 
Could this indicate a continuation of poor contractor surveillance practices previously 
noted by the GAO? 

The Source Selection category received the most attention overall by the peer- 
review teams with 115 out of 288 entries. The source selection process receives a good 
deal of emphasis in training and in practice by the contracting community, while the 
contract administration phase is sometimes considered to be neglected. An interview- 
based study with members of peer-review teams could potentially identify whether the 
Source Selection category receives so much attention in the DPAP database of peer- 
review results because it is simply better understood than the other categories. 

The peer-review process will continue to evolve at the DOD level and the service 
component level. As the process evolves, the lessons provided by the peer-review process 
should be disseminated back out to the contracting workforce. As this occurs, will 
sustained protest rates decrease? 

The service components are mandated by the peer-review policy to execute peer 
reviews at the service level for lower dollar threshold contract actions. Are the services 
complying with this requirement? Are they compiling a database similar to the DPAP 
database of peer-review results? Can the peer-review results be obtained for further 
analysis? The Army FAR supplement, the AFARS, provides more detail on the peer- 
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review requirement. The AFARS assigns responsibility for peer reviews of solicitations 
and contracts valued at greater than $50 million to a Solicitation Review Board (SRB) 
and Contract Review Board (CRB). The Principal Assistant Responsible for Contracting 
(PARC) has responsibility for establishing peer -review procedures for contracts valued 
at less than $50 million. 

The AFARS 5101.170(l)(c) also mandates that: 

The SRB/CRB will be an independent, multi-functional team comprised of 
senior level experts, which will at a minimum include representatives from 
the acquisition center, small business office, office of counsel, 
requirements community, and in the case of non-competitive actions, the 
competition advocate. (AFARS, 2014) 

From that definition, it would seem evident that the Army is attempting to garner 
insights from all angles of the acquisition process into the peer-review process. The 
AFARS also stipulates that the same members will take part in both the SRB and CRB; 
although a good idea, procurement lead-times might often prevent this from happening 
given typical military permanent change of station (PCS) cycles. There is also strict 
language on who will chair the boards, and there is no ability to delegate the authority. 
The PARC is required to chair SRB/CRB for actions valued between $50 million and 
$250 million, while the Head of the Contracting Activity (HCA) has responsibility for the 
$250 million to $1 billion range. Interestingly, per AFARS 5101.170(l)(e), PARCs and 
the HCA may waive the requirement for a formal review board. The waiver is required to 
be in writing and included in the contract file. Data on the number of peer reviews 
conducted, along with any issues or trends, best practices, or relevant feedback on the 
process itself, is required to be submitted quarterly (along with information on waivers of 
the review process) to the Procurement Policy and Support Directorate (AFARS, 2014). 

AFARS 5101.170(2) continues with further direction on the handling of post¬ 
award peer reviews. Post-award peer reviews for contracts valued at between $500 
million and $1 billion will be conducted for services contracts that were approved by a 
Headquarters, Department of the Army (HQDA)-level Army Services Strategy Panel 
(ASSP). The post-award review minimum review requirements are as follows: 

• Contract performance in terms of cost, schedule, and requirements. 
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• Use of contracting mechanisms, including the use of competition, the 
contract structure and type the definition of contract requirements, cost or 
pricing methods, the award and negotiation of task orders, and 
management and oversight mechanisms. 

• Contractor’s use, management, and oversight of subcontractors. 

• Staffing of contract management and oversight functions 

• Extent of any pass-through and excessive pass-through charges by the 
contractor (as defined in Section 852 of the National Defense 
Authorization Act for Fiscal Year 2007, Public Law 109-364). 

ASSP panel members are to address whether or not the services review structure 
is effective, evaluate the current acquisition as to adequacy of competition, provide an 
assessment of contract performance, assess the conduct of the government surveillance 
plan, and pass on any lessons learned or best practices for use on other ongoing 
acquisitions (AFARS, 2014). Additionally, the Army Contracting Command Desk Book 
(May 2012 edition), the handbook for Army contracting officers, provides the 
requirements for peer reviews listed from the AFARS. The only additional guidance it 
provides are two toolkits for pre-award and post-award peer reviews. The toolkits list 
documents required by the OSD-level peer-review team and elements required to be 
confirmed by the OSD peer-review team (ACC, 2012). 

Finally, interviews could be conducted with DPAP personnel to obtain time- 
phased data about DPAP database of peer-review results entries. The data could then be 
analyzed to determine if the competency gaps identified in the Department of Defense 
Contracting Workforce Competency Assessment Final Report (DPAP, 2007) are closing. 
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APPENDIX. FIGURES AND TABLES 
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Figure 1. DOD Acquisition Trends (from Gansler, 2007) 
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Defense Acquisition Workforce - DOD-Wide 

Distribution by Years to Retirement Eligibility (Civilians)(FY09) 


22.862 or 37.5% of the 
Civilian membersol the detente 
acquisition workforce will 
become elijibieto retire 
with full benefits before 2020. 


Retirement Eligible 

19.39S or 16% of the civilian 
members of the defense acquisition 
workforce are eligible 
to retire with full benefits. 
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Figure 2. Defense Acquisition Workforce Retirement Eligibility Distribution 

(from DOD, 2010, p.2-22) 
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Figure 3. DOD Contract Obligations and Acquisition Workforce Trends 

(from GAO, 2006, p.4) 
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DoD Criteria for the Acquisition of Services (Preiwvard) 


1 

Tenet 

Review Criteria: 

Primarily 

Assessed at: 

1 

Acquisition Strategy: Service acquisitions should have a comprehensive acquisition strategy that 
reflects program objectives, leverages spend data to arrive at strategc sourcing solutions for the 
enterprise being supported incorporates strategc contradiigtools is developed poor to the 
issuance of a solicitation (amended as applicable), and is adiered to througiout performance 

Detailed written and 
approved acquisition 
strateqv 

Review/Approval of 
Acquisition Strateqv 

2 

Clearly Defined Requirements: Service acquisitions should use a performance work statement or 

statement of objectives that clearly defines the services the program seeks to receive 

Requirements clearly 
stated 

Peer Review 

3 

Period of Performance: Contract penod of performance should reflect a length that is consistent 

with technologcal dependence industry standards and sufiaent time to reclaim program ownership 
(in cases with an acquisition history of a singe provider) such that fair competition can occur 
typically this is 3-5 years with certain exceptions (e g performance-based logistics and energy- 
savings performance contracts) Singe-award contracts forkncwledgebased services should be 
limited to three years (inducing options) unless, by exception, fully justified for longer penods 

Length of contract 
appropnate 

Review/Approval of 
Acquisition Strateqv 

4 

Appropriate Contract Type: Services acquisitions should be predsposed toward CPFF or CPIF 

arrangements when robust competition or recent competitive pnang history does not exist to build 
sufficient cost knowledge of those services within that market segment When robust competition 
exists, a there is recent competitive pnang history. FFP should typically be used Award fee 
contracts should be limited as they provide limited motivation for cost dsaplne T&M is the least 
preferable contract type and must be justified when used andlimited(e g < 10% of contract value) 

Contract type 
appropnate 

Review/Approval of 
Acquisition Strateqy 

5 

Socio-Economic Considerations: The Department higily values small business contnbutions and 

expects maximum opportunities for small business participation 

Small business 

opportunities 

Review/Approval of 

Acquisition Strateqy 

e 

Participation Decision Points: Acquisitions with longer penods of performance, particularly 

multiple award contracts, should indude decision ports (on and offramps) to ensure a qualified pool 
of contradors that will provide contiguous service througiout the life of the contrad 

Decision points (on and 
off-ramps) for longer 
term contracts 

Review/Approval of 
Acquisition Strateqy 

7 

Competition: Service acquisition requirements should be articulated in sudi a way to provide for 

maximum competition and for multple award contracts, througiout the life of the contract with 
meanmqful competition for orders Evaluation factors are tied to key proqram requirements 

Approach provides for 
robust competition 

Review/Approval of 
Acquisition Strateqy 

8 

Objective Incentives: Objedive cntena will be utilized, whenever possible, to measure contract 

performance Service contracts valued at $1 bilion or more shall indude provisions toadiieve 
productivity improvements and cost efficiencies throudiout the contract penod 

Award/Incentive Fee 

set up to reward 
effedive outcomes 

Review/Approval of 
Acquisition Strateqv 

9 

Inherently Governmental Functions Acquisitions for services must address Ihe extent of the 

agency's reliance on contractors to perform fundions closely associated with inherently 
governmental functions as required by FAR Subpart 7 5. and DFARS Subpart 207 5 

Required 
determinations 
appropnatelv executed 

Peer Review 

10 

Conflict of Interests: When one contractorwill provide oversigit for another contrador or dired 

work to another contractor, the acquisition documentation should address measures to 
reduce/elimmate potential conflid of interest 

Evaluate financial 

interest of support 
contractors as regured 

Peer Review 

11 

Performance Management: Service acquisitions should utilze performance basedcharadenstics 

to the maximum extent practicable to include measures that are directly tied to progam objedives 
These measures should include consideration of progam cost 

Meaningful 

performance measures 

Peer Review 

12 

Contract Surveillance: Service acquisitxxis must have assigned contracting officer representatives 

(CORs) that use tailored qualty assurance surveillance plansto monitor contractor performance 

Appropnate program 
oversight 

Peer Review 


Figure 4. DOD Peer-Review Criteria for Services Acquisitions during the 
Pre-award Phase (from DPAP, 2013c) 
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DoD Criteria for the Acquisition of Services (Postaward) 



Tenet 

Review Criteria: 

Primarily 

Assessed at: 

1 

Contractor Performance Assessment: Service acquisitions undergo penodic reviews to ensure 
the program is on course with respect to cost schedule and performance requirements, and any 
necessary adjustments are made 

Program meets or 
exceeds established 
cost, schedule and 
performance cntena 

Peer Review 

2 

Maintaining Competition: Multiple award service acquisrtions use contracting mechanisms, 
including the use of competition, the contract structure and type, the definition of contract 
requirements, cost or pncrg methods, the award and negotiation of task orders, and management 
and oversight mechanisms In competitive procurements (to include multiple award IDIQ competition 
for task orders), offerors must be given a reasonable amount of time to submt a proposal (typically 

30 days or more) If sufficient time is provided for offerors to respond and yet only one offer is 
received in a compettive procurement contracting officers should not depend on the standard at 

FAR 15 403-1(c)(ii) in determining the pnce to be fair and reasonable 

Robust competition fa 
orders, appropnate 
contract type well- 
defined requirements, 
appropnate cost/pnang 
methods 

Peer Review 

3 

Contractor's Subcontract Management: Service acquistions undergo penodc reviews to monitor 
the contractor's use, management and oversight of subcontractors 

Contractor's 
subcontract 
management evaluated 

Peer Review 

4 

Contract Surveillance: Service acquisitions undergo penodc reviews to ensure the appropnate 
staffinq of Government contract management andoversiqht functions to include CORs 

Appropnate 

Government oversight 
personnel in place 

Peer Review 

5 

Assessment of Excessive Pass-Through Charges: Service acquisitions undergo penodc review 

to evaluate the extent of any pass-throughs, and excessive pass-through charges (as defined in 
DFARS 252 215-7003/4). by the contractor 

Contractor's pass¬ 

through charges 
evaluated 

Peer Review 

6 

Inherently Governmental Functions: For service acquistions underwhich one contractor provides 

oversight for services performed by other contractors, penodc reviews evaluate the extent of the 
agency's reliance on the contractor to perform acquisition functions closely associated with inherently 
governmental functions asdefinedin section 2383(b)(3) of title 10. United States Code 

Evaluate appropnate 
performance of 
contracta functions (as 
required) 

Peer Review 

7 

Conflicts of Interest (Financial): For service acquisitions underwhich one contractor provides 

oversight for services performed by other contractors, penodc reviews evaluate the financial interest 
of any pnme contractor performing acquisition functions descnbed in paragraph six in any contract or 
subcontract with regard to which the contractor provided advice or recommendations to the agency 

Evaluate financial 
interest of support 
contractors (as 
required) 

Peer Review 

8 

Objective Incentives: For contracts with award and/or incentive fee. objedrve cntena will be 
utilized, whenever possible, to measure contract performance Where objective cntena exist and it 
is appropnate to also incentive subjective elements of performance the most appropnate contract 
type would be a multiple incentive type contract containing both incentive and award fee cntena 

Award/incentive fees 
are paid in a manner 
consistent with DoO 
policy, a with a 
corrective action plan n 
place to address past 
inconsistencies 

Peer Review 


Figure 5. DOD Peer-Review Criteria for Services Acquisitions during the 
Post-award Phase (from DPAP, 2013c) 
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OSO level Peer Reviews 

Best Practices, lessons learned, and Recommendations (as of December 20,2013) 


Ci»tonr 

Type of Contract 

Feedback 

Review Phase 

Type of Feedback 

Incentive and 

Award Fee 

logistical Services 
(Competitive) 

Solicitation contained provtskin for an award-term Peer Review Team (PRT) recommended adding 

language to indkate that an award of ’Excellent* does not guarantee exercise of an award term option by 

the Government 

Phase 1 

Recommendation 

Incentive and 

Award Fee 

Competitive Services 

Contract 

The planned contract structure provides for a Cost-Plus-Incentive-Fee (CP1F) arrangement with incentives 
only relating to cost. The Peer Review Team (PRT) recommends mcentivihng schedule and/or technical 
performance as wel as cost The negotiated incentive arrangement could take into account the 
contractor's projected cash flow. 

Phase 2 

Recommendation 

Incentive and 

Award Fee 

Non-Competitive Services 
Contract 

Compared to other types of contractors. Federaly Funded Research & Development Centers (FFRDCs) 
have unique fee arrangements, which are discussed tn DFARS 21S.404.7S, Fee Requirements, typicaty 
referred to as *fee-for-need.' DFARS states that FFRDCs may be provided no fee and that the decision to 
pay fee rest solely with the contracting offer. 

Phase 1 

Recommendation 

Incentive and 

Award Fee 

Competitive Multiple Award 
(Combination of firm-fixed 
price and cost type line 
items) 

Award Fee Plan (AFP) does not reflect the best business arrangement for the government in the folowing 
areas a) Roll-Over of award fee amounts; b) Provisional fees. Therefore, the acquisition team was 
encouraged to consider develop rig cost and performance incentives in beu of award fees in accordance 
with DoD pokey. 

Phase 1 

Recommendation 

Incentive and 

Award Fee 

Competitive Multiple Award 

Services Contract 

A* drafted, offerors wiB propose a fee pool percentage and the evaluation teem/ProcurWig Contracting 

Officer (PCO) must determine whether or not the offer is ‘balanced.’ The Peer Review Team (PRT) 
suggested defining the fee pool percentage and not allowing the offerors to propose a percentage. This 
ensures there is an adequate pool and simplifies the evaluation of proposals by eliminating the need to 
evaluate ‘balance.’ 

Phase 1 

Recommendation 

Incentive and 

Award Fee 

Non-Competitive Weapon 
System 

The Peer Review Team (PRT) recommended the contracting officer ensure that cost is not considered in 
both incentive fee and award fee. that maximum use of objective criteria be used to develop award fee 
enteru whenever possible, and that performance incentives are duplicated in the award fee criteria. 
Although the draft award fee plan incorporates some objective measure, there is still much vagueness in 
the wording and more measures of processes than outcomes. 

Phase 2 

Recommendation 

Incentive and 

Award Fee 

Non-Competitive Weapon 
System 

The amount or percent allocated for each performance incentive event should be gradual *i nature, so the 
fees are not front-loaded and ample fee is left through contract completion. In addition, clear definition of 
success for all events must be communicated to the contractor. For example, the definition for exit and 
entrance criteria and what comprises an IPR (In Process Review) was not noted in the attachment referred 
to in the solicitation, likewise, we recommend the Contracting Officer consider whether performing an IBR 
with 180 days, which is required by DFARS clause 2S2.234-7002(e). should be part of any performance 
incentive. If it is. the resulting fee percent allocated to it. should be minimal and the criteria for successful 
completion clearly spelled out in the resulting contract (as currently, the information relating to the IBR is 

scant) 

Phase 2 

Recommendation 


Figure 6. Excerpt From DPAP Database for OSD-Level Peer Reviews 

(from DPAP, 2013a) 


DPAP Peer 

Review Phases 

1) Prior to issuance of the soliciation 

(No comparable 
step) 

2) Prior to request for final proposal 
revisions & 3) Prior to contract award 

4) Post-award review 

Contracting 

Process Phases 

1) Procurement 
Planning 

2) Solicitation 
Planning 

3) Soliciation 

4) Source Selection 

5) Contract 
Administration 

6) Contract 
Closeout 


Figure 7. DPAP Peer-Review Phases Compared to the Contracting Process Phases 
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Market Research 
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Occurrences by Category Type 



0 20 40 60 80 100 120 


■ Occurrences 


Figure 8. Frequency Distribution of Category Type (after DPAP, 2013a) 
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Occurrences by Contract Type 


Weapon System Technology Demonstration 
Weapon System Production Lot Buy (Sole Source) 
Weapon System Development Sole Source 
Weapon System Development 
Sole Source Procurement of Services 
Non Competitive Weapon System Development 
Non Competitive Weapon System 
Non Competitive Supplies Contract 
Non Competitive Services Contract 
Non Competitive Multiple Award Services Contract 
Multiple Award Contract for R&D 
Multiple Award Construction Contract 
Logistical Services (Competitive) 
Design/Build Construction Contract 
Competitive Weapons System Development 
Competitive Supplies Contract 
Competitive Services Contract 
Competitive Procurement of Supplies 
Competitive Procurement of Services 
Competitive Multiple Award IDIQ Supplies Contract 
Competitive Multiple Award IDIQ 
Competitive Multiple Award Services Contract 
Competitive Multiple Award IDIQ Services 
Competitive Multiple Award IDIQ R&D Contract 
Competitive Multiple Award (Combination Multiple... 

Competitive IQC 
Competitive IDIQ Services Contract 
Competitive IDIQ for IT Products 
Competitive IDIQ 
Competitive Design/Build FPIF 
Commodity (Competitive) 
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Figure 9. Frequency Distribution of Contract Type (after DPAP, 2013a) 
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Occurrences by Review Phase 



i Occurrences 


Figure 10. Frequency Distribution of Review Phase (after DPAP, 2013a) 


Occurrences by Feedback Type 


Lessons Learned 


Best Practice 


Recommendation 



■ Occurrences 


Figure 11. Frequency Distribution of Type of Feedback (after DPAP, 2013a) 
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Recommendation Occurrences By Phase 



Figure 12. Frequency Distribution of Recommendations by Phase 

(after DPAP, 2013a) 


Occurrences of Recommendation 
Within Categories 


Terms & Conditions 
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Post Award Admin 
Incentive and Award Fee 
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Figure 13. Frequency Distribution of Recommendations within Categories 

(after DPAP, 2013a) 
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Occurrences of Recommendation Type Feedback by 
Contract Type 


Weapon System Technology Demonstration 
Weapon System Production Lot Buy (Sole Source) 
Weapon System Development Sole Source 
Weapon System Development 
Sole Source Procurement of Services 
Non Competitive Weapon System Development 
Non Competitive Weapon System 
Non Competitive Supplies Contract 
Non Competitive Services Contract 
Non Competitive Multiple Award Services Contract 
Multiple Award Contract for R&D 
Multiple Award Construction Contract 
Logistical Services (Competitive) 
Design/Build Construction Contract 
Competitive Weapons System Development 
Competitive Supplies Contract 
Competitive Services Contract 
Competitive Procurement of Supplies 
Competitive Procurement of Services 
Competitive Multiple Award IDIQ Supplies Contract 
Competitive Multiple Award IDIQ 
Competitive Multiple Award Services Contract 
Competitive Multiple Award IDIQ Services 
Competitive Multiple Award IDIQ R&D Contract 
Competitive Multiple Award (Combination Multiple Award... 

Competitive IQC 
Competitive IDIQ Services Contract 
Competitive IDIQ for IT Products 
Competitive IDIQ 
Competitive Design/Build FPIF 
Commodity (Competitive) 



■ Occurrences of Recommendation 


Figure 14. Frequency Distribution of Recommendations by Contract Type 

(after DPAP, 2013a) 
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Table 1. Percentage of Recommendations by Category (after DPAP, 2013a) 


Categories 

Total 

Occurrences 

Occurrences of 
Recommendation 

Percent of 
Recommendation 

Incentive and Award Fee 

23 

20 

86.95652174 % 

Market Research 

4 

0 

0% 

Peer Review 

1 

0 

0% 

Post-award Admin 

35 

9 

25.71428571 % 

Pricing 

46 

41 

89.13043478 % 

Requirements/PW S/ 
SOW 

24 

17 

70.83333333 % 

Source Selection 

115 

66 

57.39130435 % 

Terms & Conditions 

40 

37 

92.5 % 

Total 

288 

190 

65.97222222 % 
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Table 2. Percentage of Recommendations by Contract Type (after DPAP, 2013a) 


Contract Type 

Total 

Occurrences of 
Recommendation 

Percentage of 
Recommendation 

Commodity (Competitive) 

4 

3 

75 

Competitive Design/Build FPIF 

2 

2 

100 

Competitive IDIQ 

1 

1 

100 

Competitive IDIQ for IT Products 

4 

4 

100 

Competitive IDIQ Services Contract 

6 

2 

33.33333333 

Competitive IQC 

1 

1 

100 

Competitive Multiple Award (Combination 
Multiple Award FFP and Cost type line items) 

7 

7 

100 

Competitive Multiple Award IDIQ R&D Contract 

1 

0 

0 

Competitive Multiple Award IDIQ Services 

22 

3 

13.63636364 

Competitive Multiple Award Services Contract 

59 

41 

69.49152542 

Competitive Multiple Award IDIQ 

5 

2 

40 

Competitive Multiple Award IDIQ Supplies 

2 

1 

50 

Contract 


Competitive Procurement of Services 

20 

9 

45 

Competitive Procurement of Supplies 

14 

9 

64.28571429 

Competitive Services Contract 

39 

16 

41.02564103 

Competitive Supplies Contract 

1 

1 

100 

Competitive Weapons System Development 

16 

14 

87.5 

Design/Build Construction Contract 

3 

3 

100 

Logistical Services (Competitive) 

10 

7 

70 

Multiple Award Construction Contract 

2 

2 

100 

Multiple Award Contract for R&D 

6 

5 

83.33333333 

Non-Competitive Multiple Award Services 

11 

11 

100 

Contract 

Non-Competitive Services Contract 

4 

4 

100 

Non-Competitive Supplies Contract 

10 

8 

80 

Non-Competitive Weapon System 

8 

8 

100 

Non-Competitive Weapon System Development 

6 

5 

83.33333333 

Sole-Source Procurement of Services 

1 

1 

100 

Weapon System Development 

5 

4 

80 

Weapon System Development Sole Source 

1 

0 

0 

Weapon System Production Lot Buy (Sole Source) 

3 

3 

100 

Weapon System Technology Demonstration 

14 

13 

92.85714286 

Total 

288 

190 

65.97222222 
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Table 3. 


DOD Contracting Competency Model (after DPAP, 2007) 


DoD Contracting Comp**l*»ncy Modt»l 
(rtgklUhrd March 2007 
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